Accessory and information processing system

ABSTRACT

An example accessory can be attached to an information processing device. The accessory includes a housing, a movable operation section, and a movable portion. An inside of the housing is visible from an outside thereof through at least a portion thereof. At least a portion of the operation section is exposed on the outside of the housing. The movable portion is placed at a position inside the housing that is visible from the outside of the housing through the portion of the housing. A position and/or an attitude of the movable portion changes in response to the operation section being operated.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2015-024052, filed onFeb. 10, 2015, is herein incorporated by reference.

FIELD

The present technique relates to an information processing systemincluding an information processing device and an accessory that can beattached to the information processing device, and relates to anaccessory, an information processing device, a storage medium storing aninformation processing program, an operation determination method and aprocess performing method, for use in the information processing system.

BACKGROUND AND SUMMARY

There are conventional accessories which, when connected to a portabledevice, add functions to the portable device. For example, there aretechniques where an external controller as an accessory is connected toa connector provided on a controller of a game device. This enables awider variety of game operations using two controllers.

Conventional accessories have a circuit configuration for communicatingwith a portable device or a circuit configuration for performing aninformation process for generating information to be transmitted to aportable device, and there has been room for improvement in simplifyingthe configuration of the accessories.

Thus, the present specification discloses an accessory, an informationprocessing device, an information processing system, a storage mediumstoring an information processing program, an operation determinationmethod and a process performing method, with which it is possible tosimplify the configuration of the accessory.

(1)

An example accessory described herein can be attached to an informationprocessing device. The accessory includes a housing, an operationsection, and a movable portion. An inside of the housing is visible froman outside thereof through at least a portion thereof. The operationsection is a movable operation section at least a portion of which isexposed on the outside of the housing. The movable portion is placed ata position inside the housing that is visible from the outside of thehousing through the portion of the housing, and a position and/or anattitude of the movable portion changes in response to the operationsection being operated.

(2)

A portion of the housing may have an opening. The movable portion may beplaced at a position inside the housing that is visible from the outsideof the housing through the opening portion of the housing.

(3)

The movable portion may include an identification portion in an areathat is visible from the outside of the housing through the portion ofthe housing, wherein the identification portion is distinguishable fromother portions.

(4)

At least a portion of the area of the movable portion that is visiblefrom the outside of the housing through the portion of the housing maybe formed by a member that can be identified by an infrared camera.

(5)

The accessory may further include a connecting portion for connectingthe housing with the information processing device so that the portionof the housing is covered by the information processing device to whichthe accessory is attached.

(6)

The movable portion may move, in response to the operation section beingoperated, in at least one of an up-down direction, a left-rightdirection and a front-rear direction as seen through the portion of thehousing.

(7)

The operation section and the movable portion may be configured as anintegral part.

(8)

On the inside of the housing, a predetermined marker may be provided ata position inside the housing that is visible from the outside of thehousing through the portion of the housing, wherein the position atwhich the movable portion is provided is different from the position atwhich the marker is provided.

(9)

The operation section may include a first operation member and a secondoperation member that can be operated independently of each other. Thefirst operation member and the second operation member may be arrangedin line with each other in a front-rear direction as seen through theportion of the housing. The movable portion may include a first movableportion of which a position and/or an attitude changes in response tothe first operation member being operated, and a second movable portionof which a position and/or an attitude changes in response to the secondoperation member being operated. The first movable portion may bedisplaced from the second movable portion as seen through the portion ofthe housing.

(10)

The operation section may be configured to be rotatable. The movableportion may rotate in accordance with rotation of the operation section,and a shape and/or a design pattern of the movable portion as seenthrough the portion of the housing changes as the movable portionrotates.

(11)

An example information processing system described herein is aninformation processing system including an information processing devicehaving a camera, and an accessory which can be attached to theinformation processing device.

The accessory includes a housing, a movable operation section, and amovable portion. An inside of the housing is visible from an outsidethereof through at least a portion thereof. The operation section is amovable operation section at least a portion of which is exposed on theoutside of the housing. The movable portion is placed at a positioninside the housing that is visible from the outside of the housingthrough the portion of the housing, wherein a position and/or anattitude of the movable portion changes in response to the operationsection being operated. The accessory is attached to the informationprocessing device at such a position that the camera can capture animage of the inside of the housing through the portion of the housing.

The information processing device includes an operation determinationunit and a process performing unit. The operation determination unitdetermines an operation performed on the operation section based on acamera image captured by the camera and including an image of themovable portion. The process performing unit performs a predeterminedprocess based on the operation determined by the operation determinationunit.

(12)

The operation determination unit may determine whether or not theoperation section is operated based on the camera image. The processperforming unit may perform a predetermined process based on whether ornot the operation section is operated.

(13)

The operation determination unit may calculate an amount of operationperformed on the operation section based on the camera image. Theprocess performing unit may perform a predetermined process based on theamount of operation.

(14)

On the inside of the housing, a predetermined marker may be provided ata position inside the housing that is visible from the outside of thehousing through the portion of the housing, wherein the position atwhich the movable portion is provided is different from the position atwhich the marker is provided. The operation determination unit mayrecognize the predetermined marker in the camera image to determine anoperation performed on the operation section based on at least one of aposition, a direction and a shape of an image of the movable portionwith respect to the marker in the camera image.

(15)

Another example information processing system described herein is aninformation processing system including an information processing devicehaving a camera, and an accessory which can be attached to theinformation processing device. The accessory includes a housing of whichan inside is visible from an outside thereof through at least a portionthereof. On the inside of the housing, a predetermined marker isprovided at a position inside the housing that is visible from theoutside of the housing through the portion of the housing. The accessorycan be attached to the information processing device at such a positionthat the camera can capture an image of the inside of the housingthrough the portion of the housing.

The information processing device includes a marker recognition unit anda process performing unit. The marker recognition unit recognizes thepredetermined marker in the camera image obtained by the camera. Theprocess performing unit performs a predetermined process based on themarker recognition result.

(16)

The process performing unit may perform a different process depending ona type of the marker recognized.

Note that when a predetermined marker is recognized, the processperforming unit may launch a predetermined application program withwhich the accessory can be used. If a predetermined marker stops beingrecognized while a predetermined application with which the accessorycan be used is running on the information processing device, the processperforming unit may halt the execution of the application program. If apredetermined marker is recognized, the process performing unit mayperform a process of accessing a predetermined external device to obtaina predetermined application program with which the accessory can beused. If a predetermined marker is recognized, the process performingunit may perform a process of accessing an external device to obtain apredetermined web page, and displaying the web page on the displaydevice of the information processing device.

The marker recognition unit may determine whether or not a predeterminedmarker has been recognized in the camera image. When it is determinedthat a predetermined marker has been recognized, the process performingunit may determine that the accessory has been attached to theinformation processing device. Note that if it is determined that apredetermined marker has been recognized, the process performing unitmay determine that an authentic accessory has been attached to theinformation processing device and allow a predetermined informationprocess to be performed on the information processing device.

Also disclosed herein is a portable information processing device orinformation processing system which has a camera and to which theaccessory set forth above can be attached. Also disclosed herein is aninformation processing system including the accessory and theinformation processing device set forth above. Moreover, the presentspecification also discloses a storage medium storing an informationprocessing program for instructing a computer of an informationprocessing system to function as the various units set forth above, andan operation determination method or a process performing method to becarried out on the information processing system set forth above.

With the accessory, the information processing system, the informationprocessing device, the storage medium storing an information processingprogram, the operation determination method, and the process performingmethod set forth above, it is possible to provide an accessory with asimple configuration.

These and other objects, features, aspects and advantages will becomemore apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front view showing a non-limiting example portable device ofthe present embodiment;

FIG. 2 is a right side view showing a non-limiting example portabledevice;

FIG. 3 is a back view showing a non-limiting example portable device;

FIG. 4 is a block diagram showing an internal configuration of anon-limiting example portable device;

FIG. 5 shows how a non-limiting example portable device is used;

FIG. 6 shows the external appearance of a non-limiting example operationdevice;

FIG. 7 shows an example internal configuration of the operation deviceshown in FIG. 6;

FIG. 8 shows an example operation device before and after it is attachedto a portable device;

FIG. 9 shows an example configuration where an operation device isattached to a portable device;

FIG. 10 shows an example of how operation buttons and movable portionsmove;

FIG. 11 shows an example captured image obtained by an infrared camera;

FIG. 12 shows another example captured image obtained by an infraredcamera;

FIG. 13 is a flow chart showing an example of the flow of a processperformed by a portable device;

FIG. 14 is a flow chart showing an example of the flow of a processperformed by an application program of a portable device;

FIG. 15 shows an example rotational operation section of an operationdevice;

FIG. 16 shows another example rotational operation section of anoperation device;

FIG. 17 shows another example of an unfolded diagram of a side surfaceof a rotatable portion;

FIG. 18 shows an example captured image obtained by an infrared cameraaccording to the present variation; and

FIG. 19 is a flow chart showing an example of the flow of a processperformed by a portable device according to the present variation.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

An accessory, an information processing system, an informationprocessing device, an information processing program, an operationdetermination method and a process performing method of the presentembodiment will now be described. The information processing system ofthe present embodiment includes a portable device, which is an exampleof the information processing device, and an operation device, which isan example of the accessory (see FIG. 8). In the present embodiment, theoperation device can be attached to the portable device, and theoperation device can be used for performing an operation on the portabledevice. That is, it is possible, with the operation device, to add anoperation function to the portable device.

[1. Configuration of Portable Device]

First, referring to FIG. 1 to FIG. 5, an example of a configuration of aportable device will be described. FIG. 1 is a front view showing anexample of a portable device of the present embodiment. FIG. 2 is aright side view showing an example of a portable device. FIG. 3 is aback view showing an example of a portable device. In the presentembodiment, a portable device 1 is a portable information processingdevice that can be held in a hand and operated by a user. For example,the portable device 1 may be a hand-held device such as a portable gamedevice, a portable telephone, a smartphone, a tablet terminal, a camera,a game controller or a mouse, or may be a terminal that can be worn by auser such as a wristwatch-shaped terminal.

As shown in FIG. 1, the portable device 1 includes a display 2, a touchpanel 3, an infrared camera 4, a distance measuring sensor 5, inputbuttons 6 (6A to 6D), an illuminating section 7 and a projector 8, whichare accommodated in a housing 10. The housing 10 (the portable device 1)has a plate-like shape, and is sized so that it can be held by a userwith one hand or both hands.

For example, the display 2 may be a liquid crystal display device, anorganic EL display device, or the like, or may by any other suitabledisplay device. The screen of the display 2 is provided so as to beexposed on the front surface (T5 surface) of the housing 10. The touchpanel 3 is provided on the screen of the display 2 for detecting theposition at which the screen has been touched by a user. The touch panel3 may be a touch panel capable of one-point detection or a touch panelcapable of multi-point detection, and may be of any suitable type suchas capacitive or resistive, for example.

The input buttons 6A to 6D each receive an input (pressing of a button)by a user. The input buttons 6A to 6D are each provided at such aposition that it can be reached by a finger of a user when the userholds the opposing sides of the portable device 1. Specifically, theinput buttons 6A and 6C are located so that they can be reached byfingers of the right hand when the user holds the portable device 1 withthe right hand, wherein the input button 6A is provided at such aposition that it can be reached by the thumb of the right hand, and theinput button 6C is provided at such a position that it can be reached bythe index finger or the middle finger of the right hand. The inputbuttons 6B and 6D are located so that they can be reached by fingers ofthe left hand when the user holds the portable device 1 with the lefthand, wherein the input button 6B is provided at such a position that itcan be reached by the thumb of the left hand, and the input button 6D isprovided at such a position that it can be reached by the index fingeror the middle finger of the left hand. As shown in FIG. 1, the inputbuttons 6A and 6B are provided on the front surface (T5 surface) of thehousing 10, and the input buttons 6C and 6D are provided on the topsurface (T4 surface) of the housing 10. Note that input sections forreceiving inputs from a user may include, in addition to the inputbuttons 6A to 6D, a cross-shaped key, an analog stick, or the like, fordirection inputs.

The infrared camera 4 includes a lens, and a sensor capable of sensinglight (infrared rays; specifically, near infrared rays). The sensor ofthe infrared camera 4 is an image sensor in which elements capable ofsensing infrared rays are arranged in a matrix, and the elements of theimage sensor receive and convert infrared rays into electric signals,outputting a two-dimensional infrared image.

The distance measuring sensor 5 emits light (e.g., infrared light) froma light source provided in the distance measuring sensor 5 and receivesthe light having been reflected off an object by means of alight-receiving element, thereby measuring the distance to the object.The distance measuring sensor 5 may be of any type, including atriangulation type and a TOF (Time Of Flight) type. The light source ofthe distance measuring sensor 5 may be an LED, a laser diode, or thelike, emitting infrared light in a particular direction.

The illuminating section 7 outputs infrared rays at a predetermined timeinterval (e.g., an interval of 1/60 sec). The illuminating section 7outputs infrared rays in sync with the timing with which the infraredcamera 4 captures images. The illuminating section 7 outputs infraredrays onto a predetermined range in the right side surface direction ofthe portable device 1. Infrared rays output by an illuminating section 7are reflected by an object, and the reflected infrared rays are receivedby the infrared camera 4, thereby obtaining an image of the infraredrays. Note that the illuminating section 7 may be used for capturing aninfrared image by the infrared camera 4 and for measuring the distanceby the distance measuring sensor 5. That is, the infrared light from theilluminating section 7 may be used both for capturing an image by meansof the infrared camera 4 and for measuring the distance by means of thedistance measuring sensor 5.

The projector 8 includes a light source for emitting visible light, andprojects text, an image, etc., onto a projection plane (a screen or thehand of a user as will be described later) by using light from the lightsource.

The infrared camera 4, the distance measuring sensor 5, the illuminatingsection 7 and the projector 8 are provided on a side surface of thehousing 10 (e.g., the right side surface: T1 surface). Specifically, theimage-capturing direction (optical axis) of the infrared camera 4 isoriented vertical to the right side surface. The detecting direction ofthe distance measuring sensor 5 and the direction in which the projector8 outputs light are also vertical to the right side surface. That is,when a user holds the portable device 1 with the left hand, the infraredcamera 4 captures an image of the space in the right side surfacedirection of the portable device 1, and the distance measuring sensor 5measures the distance to an object existing in the space in the rightside surface direction of the portable device 1. The projector 8projects an image, or the like, by outputting visible light in the samedirection as the infrared camera 4 and the distance measuring sensor 5.

An outside camera 9 is provided on the back surface (T6 surface) of theportable device 1 (FIG. 3). The outside camera 9 is typically capable ofcapturing an image in the direction vertical to the image-capturingdirection of the infrared camera 4, and is capable of capturing an imagein the direction vertical to the back surface. The outside camera 9includes a lens, and an image sensor capable of sensing visible light.The outside camera 9 captures an image of the space in the back surfacedirection, obtaining a color image (RGB image). Note that a camera maybe provided on the front surface in addition to the outside camera 9 onthe back surface, or a camera may be provided on the front surface (thesurface on which the screen of the display 2 is provided) withoutproviding the outside camera 9 on the back surface.

FIG. 4 is a block diagram showing an example of an internalconfiguration of the portable device 1. As shown in FIG. 4, in additionto the various sections mentioned above, the portable device 1 includesa vibrator 11, a microphone 12, a speaker 13, a control section 14, acommunication section 15, an attitude detection section 16, a GPSreceiver 17 and a geomagnetic sensor 18. The portable device 1 alsoincludes a battery (not shown), and receives power supply from thebattery. These sections are accommodated in the housing 10.

The control section 14 is connected to, and controls, various sectionsincluding the display 2, the touch panel 3, the infrared camera 4, thedistance measuring sensor 5, the input button 6, the illuminatingsection 7, the projector 8, the vibrator 11, the microphone 12, thespeaker 13, the communication section 15, the attitude detection section16, the GPS receiver 17 and the geomagnetic sensor 18.

Specifically, a control section 14 includes a CPU, a memory, etc., forperforming a predetermined process based on a predetermined program(e.g., an application program for performing game processes, imageprocesses, and various arithmetic operations) stored in a storage device(not shown) (e.g., a non-volatile memory or a hard disk) provided in theportable device 1. For example, the control section 14 may obtain andanalyze an image from the infrared camera 4, calculate the distance toan object based on the signal from the distance measuring sensor 5, orperform a process in accordance with input signals from the touch panel3 and the input button 6. Then, the control section 14 generates animage based on the results of the predetermined process, and outputs theimage on the display 2. Note that the program for performing thepredetermined process may be downloaded from outside via thecommunication section 15.

The vibrator 11 operates based on an instruction from the controlsection 14 and vibrates the entire portable device 1. The vibrator 11 isprovided at a predetermined position such that vibrations are easilytransmitted to the hand of a user (e.g., in the central portion, or at aposition shifted left or right therefrom, inside the housing 10).

The microphone 12 and the speaker 13 are used for inputting/outputtingsounds. The communication section 15 is used for communicating withother devices by a predetermined communication scheme (e.g., wirelessLAN, etc.). The attitude detection section 16 is, for example, anacceleration sensor or an angular velocity sensor for detecting theattitude of the portable device 1.

The GPS receiver 17 receives a signal from a GPS (Global PositioningSystem) satellite, and the portable device 1 can calculate the positionof the portable device 1 itself by receiving the signal. For example,when a predetermined operation is performed at a particular position(e.g., a gesture input or a button input using the infrared camera 4,shaking the portable device 1, etc., to be described later), theportable device 1 may display an object associated with the particularposition. For example, where a game is played on the portable device 1,and the portable device 1 is at a particular position, an objectassociated with the particular position may appear in the game.

The geomagnetic sensor 18 is a sensor capable of detecting the directionand the magnitude of a magnetic field. For example, based on thedetection result of the geomagnetic sensor 18, the portable device 1 maydetermine whether it is facing a particular azimuthal direction, and theportable device 1 may display an object when a predetermined operation(e.g., a gesture input mentioned above) in a particular azimuthaldirection. For example, where a game is played on the portable device 1,an object associated with a particular azimuthal direction may appear inthe game. The portable device 1 may combine together GPS informationobtained by the GPS receiver 17 and azimuthal direction informationobtained by the geomagnetic sensor. For example, if the portable device1 is at a particular position and is facing a particular azimuthaldirection, the portable device 1 may display an object associated withthe particular position and the particular azimuthal direction, or suchan object may appear in the game.

FIG. 5 shows an example of how the portable device 1 is used. As shownin FIG. 5, a user can hold the portable device 1 with one hand and makegesture inputs using the other hand. Note that a gesture input may beany input made with an operation object that is operated (moved) by auser. An operation object may be the body of the user (which may be thewhole body or a part of the body such as a hand or the face), an objectheld by the user, or may be both the body and an object. The portabledevice 1 may recognize the shape of the operation object as a gestureinput, may recognize the movement of the operation object as a gestureinput, or may recognize a combination of the shape and the movement as agesture input. For example, a user may make a gesture input by using theshape, the movement, the position (with respect to the portable device1), the orientation (attitude), etc., of the hand.

In the present embodiment, the portable device 1 captures an image(infrared image) by means of the infrared camera 4, and determines agesture input based on the captured image (referred to also as thecamera image). Specifically, when a captured image is obtained from theinfrared camera 4, the portable device 1 calculates the shape, theposition, and/or the orientation (attitude) of the object based on thecaptured image obtained. Note that based on a plurality of capturedimages obtained through repetition, the portable device 1 may calculatethe movement of the object (the change of shape, the moving direction,the moving speed, etc.). The portable device 1 determines a gestureinput based on these calculation results.

[2. Configuration of Operation Device]

Next, referring to FIG. 6 and FIG. 7, an example configuration of anoperation device will be described. FIG. 6 shows the external appearanceof an example operation device. In the present embodiment, an operationdevice 20 can be attached/detached to/from the portable device 1, andincludes operation sections (buttons 22A to 22D). That is, the operationdevice 20 is an accessory that adds an operation function to theportable device 1. The portable device 1 detects an operation performedon the operation device 20, and performs a process in accordance withthe detected operation. Here, the operation is detected by using theinfrared camera 4, the details of which will be described later.Therefore, the operation device 20 needs no electrical configuration,such as an electronic circuit, for detecting the operation and fortransmitting the detection result to the portable device 1. Therefore,according to the present embodiment, it is possible to simplify theconfiguration of the operation device 20. The details will now bedescribed.

As shown in FIG. 6, the operation device 20 includes a housing 21. Thehousing 21 is attached to the portable device 1 in such a manner thatthe housing 21 can be attached/detached to/from the portable device 1.That is, the operation device 20 includes a connecting portion (notshown) for connecting the housing 21 (the operation device 20) with theportable device 1. Note that the connecting portion may have anyconfiguration, and may be a conventional mechanism.

For example, the connecting portion may include a claw to be engagedwith a predetermined portion of the portable device 1 (e.g., aprotruding portion formed along the periphery of the front surface (T5surface)), and the housing 21 may be secured on the portable device 1with the claw engaged with the predetermined portion. For example, theconnecting portion may be such a mechanism that the housing 21 issecured on the portable device 1 with a portion of the portable device 1being clamped in the up-down direction. The connecting portion may beconnected to the portable device 1 magnetically. For example, theoperation device 20 may be connected to the portable device 1 by virtueof the attraction between a magnet of the portable device 1 and a magnetof the connecting portion. That is, “to attach” as used herein meansthat the positional relationship between the two members is somewhatfixed, and the connection between the two members is not limited to amechanical connection, but may include a connection between the twomembers by virtue of a magnetic force. In other embodiments, theportable device 1 and the operation device 20 may be connected togethervia a connection member. That is, as the portable device 1 and theconnection member are connected together and the connection member andthe operation device 20 are connected together, the positionalrelationship between the portable device 1 and the operation device 20may be fixed.

In the present embodiment, the housing 21 (the operation device 20) isattached to the portable device 1 so fixedly that the attachment cannotbe removed by up to a certain amount of force. The housing 21 (theoperation device 20) is detached from the portable device 1 when astronger force is applied thereto.

As shown in FIG. 6, a hole (opening portion) 21 a is formed in thehousing 21. Thus, the operation device 20 is configured so that theinside (the inner side) of the housing 21 can be seen through the hole21 a. Note that the housing 21 may have any configuration such that theinside of the housing 21 is visible from the outside through apredetermined portion of the housing 21. For example, in otherembodiments, the predetermined portion may be formed by a transparentmember instead of the provision of the hole 21 a in the predeterminedportion.

Here, in the present embodiment, a reference marker 21 d is provided ona wall surface on the inside of the housing 21 (the inner wall surfaceof a surface 21 c in FIG. 6). Movable portions 23A to 23D to bedescribed later are provided on the inside of the housing 21 (see FIG.7). The hole 21 a is formed at such a position that the movable portions23A to 23D and the reference marker 21 d are visible when one sees theinside of the housing 21 from the outside of the housing 21 through thehole 21 a.

Note that the reference marker 21 d is used as a reference forcalculating the positions of the movable portions 23A to 23D, thedetails of which will be described later. The reference marker 21 d isformed at a predetermined position on the inside of the housing 21. Theshape of the reference marker 21 d is predetermined. The referencemarker 21 d is of a material (color) such that the infrared camera 4 candistinguish the reference marker 21 d from other portions (other thanthe reference marker 21 d) of the wall surface on the inside of thehousing 21. The reference marker 21 d may be a design pattern providedon the inner surface of the housing 21 or a member placed on the innersurface of the housing 21.

The operation device 20 includes operation buttons 22A to 22D, which areexample operation sections. Note that where the operation buttons 22A to22D are not distinguished from each other, the designation “operationbutton 22” may be used to refer to any one of the operation buttons 22Ato 22D. The operation buttons 22A to 22D are provided so as to beexposed on a predetermined surface of the housing 21 (a button surface21 b in FIG. 6). The operation buttons 22A to 22D are projecting fromthe button surface 21 b, and can be pressed toward the inside of thehousing 21. Specifically, each operation button 22 is supported, in thehousing 21, with respect to the housing 21 while being urged upwardly bya biasing member (e.g., an elastic body such as a spring) (not shown).The operation button 22 moves downward by being operated (pressed) bythe user.

FIG. 7 shows an example internal configuration of the operation deviceshown in FIG. 6. Note that FIG. 7 shows only a part of the configurationinside the housing 21. As shown in FIG. 7, the operation device 20includes the movable portions 23A to 23D provided inside the housing 21.Note that where the movable portions 23A to 23D are not distinguishedfrom each other, the designation “movable portion 23” may be used torefer to any of the movable portions 23A to 23D.

As shown in FIG. 7, the movable portions 23A to 23D are connected to theoperation buttons 22A to 22D, respectively. That is, the movable portion23A is connected to the operation button 22A, the movable portion 23B tothe operation button 22B, the movable portion 23C to the operationbutton 22C, and the movable portion 23D to the operation button 22D. Themovable portion 23 moves (downward in the present embodiment) inresponse to an operation on the operation button 22 connected thereto.In the present embodiment, each operation button and a movable portionconnected thereto are configured as an integral part. Note however thatthe movable portion may be configured so as to be moved in accordancewith the movement of the operation button, and an operation button and amovable portion connected thereto may be separate parts in otherembodiments.

In the present embodiment, at least a portion of the movable portion 23(the hatched area shown in FIG. 7: hereinafter referred to as a“recognition portion”) is of a material (color) such that the infraredcamera 4 can distinguish this portion from other portions (other thanthe portion of the reference marker 21 d) of the wall surface on theinside of the housing 21. The recognition portion may be of aretroreflective material, for example. Then, the infrared light from theilluminating section 7 is reflected more toward the infrared camera 4,making it easier for the infrared camera 4 to recognize the recognitionportion. This also makes it easier for the infrared camera 4 torecognize a recognition portion that is located away from the infraredcamera 4. The movable portion 23 is connected to the lower portion ofthe operation button 22. Here, for each of the movable portions 23A to23C, the recognition portion is placed under (directly below) thecorresponding operation button 22. For the movable portion 23D, on theother hand, the recognition portion is displaced from a position that isdirectly below the corresponding operation button 22D (see FIG. 7). Thatis, the recognition portions of the movable portions 23A to 23D arearranged so that the recognition portions are visible through the hole21 a.

Note that the shape and the size of the housing 21, and the shape, thesize and the number of the operation button 22 and the movable portions23 are not limited to any particular shape, size and number. The shape,the size and the number of these members shown in FIG. 6 and FIG. 7 areillustrative, and the operation device in other embodiments may have adifferent configuration from that shown in FIG. 6 and FIG. 7.

[3. Attachment of Operation Device to Portable Device]

Next, referring to FIG. 8 and FIG. 9, the attachment of the operationdevice 20 to the portable device 1 will be described. In the presentembodiment, the operation device 20 can be attached at least to a sidesurface (more specifically, the T1 surface) of the portable device 1.

FIG. 8 shows an example of the operation device 20 before and after itis attached to the portable device 1. As shown in FIG. 8(b), theoperation device 20 is attached to the portable device 1 at such aposition that the hole 21 a faces the infrared camera 4 (and theilluminating section 7). Therefore, with the operation device 20attached to the portable device 1, the infrared camera 4 can capture animage of the inside of the housing 21. In this position, theilluminating section 7 can output infrared light into the inside of thehousing 21. In the present embodiment, since infrared light is output bythe illuminating section 7 into the inside of the housing 21, it ispossible with the infrared camera 4 to capture an image of the inside ofthe housing 21 even if the ambient light does not enter the inside ofthe housing 21. Note that in the present embodiment, the housing 21 isof a material that does not allow light (at least infrared light) topass therethrough, except for the hole 21 a (or a portion of atransparent member provided instead of the hole 21 a).

As shown in FIG. 8(b), with the operation device 20 attached to theportable device 1, the hole 21 a is covered by the operation device 20.That is, the connecting portion connects the housing 21 with theportable device 1 so that the hole 21 a of the housing 21 is covered bythe portable device 1 to which the operation device 20 is attached.Then, since the infrared camera 4 does not (substantially) detectambient light, it is possible to reduce the possibility of falserecognition due to ambient light (false recognition of the referencemarker 21 d and/or the movable portion 23). That is, it is possible tomore accurately recognize the inside of the housing 21 based on thecaptured image obtained by the infrared camera 4.

FIG. 9 shows an example configuration where the operation device 20 isattached to the portable device 1. When the operation device 20 isattached to the portable device 1, the user may grip a left-side portionof the portable device 1 with the left hand and grip the operationdevice 20 with the right hand. Then, the user can operate the operationbuttons 22A to 22D of the operation device 20. Note that the operationdevice 20 may be attached to the portable device 1 so that the buttonsurface 21 b of the operation device 20 is located generally on the sameplane (i.e., generally at the same position with respect to the z-axisdirection) as the front surface (T5 surface) of the portable device 1.Then, the user can operate the portable device 1 and the operationdevice 20 while having both hands at the same position with respect tothe z-axis direction, thus making it easier to perform operations.

Note that in other embodiments, the operation device 20 may beconfigured so that the operation device 20 cannot be attached to theportable device 1 at such a position that the hole 21 a is not facingthe infrared camera 4 and the illuminating section 7 (i.e., at such aposition that the infrared camera 4 cannot capture an image of theinside of the housing 21, and/or the illuminating section 7 cannotoutput light into the inside of the housing 21). For example, thehousing 21 may include a protrusion that fits into a depression on theportable device 1 when the housing 21 is placed at an appropriateposition (a position such that the hole 21 a faces the infrared camera 4and the illuminating section 7) (the protrusion of the housing 21 andthe depression of the portable device 1 may be reversed). Then, thealignment can be made with the protrusion of the housing 21 and thedepression of the portable device 1, and it is therefore possible toattach the operation device 20 to the portable device 1 at anappropriate position. Note that in other embodiments, the operationdevice 20 may be allowed to be attached to other positions of theportable device 1. For example, the operation device 20 may be allowedto be attached to the portable device 1 at such a position that the hole21 a faces the outside camera 9.

[4. Determining Operation Performed on Operation Device]

Next, referring to FIG. 10 to FIG. 12, how an operation performed on theoperation device 20 is determined by the portable device 1 will bedescribed. When the operation device 20 is attached to the portabledevice 1, the infrared camera 4 captures an image of the movableportions 23A to 23D inside the housing 21 through the hole 21 a. Now,the movable portion 23 moves in response to an operation performed onthe corresponding operation button 22. Therefore, by calculating theposition of the movable portion 23 in the captured image, it is possibleto determine that the operation button 22 is operated. The details ofthe method for determining an operation will now be described.

FIG. 10 shows an example of how the operation button 22 and the movableportion 23 move. FIG. 10 is a diagram showing the operation buttons 22Ato 22D and the movable portions 23A to 23D as seen from theimage-capturing direction of the infrared camera 4 (i.e., the directionof viewing the inside of the housing 21 from the outside of the housing21 through the hole 21 a). In the present embodiment, when the operationbutton 22A is pressed, the movable portion 23A moves downward (thenegative z-axis direction) ((1) shown in FIG. 10). Similarly, when theoperation button 22B is pressed, the movable portion 23B moves downward((2) shown in FIG. 10); when the operation button 22C is pressed, themovable portion 23C moves downward ((3) shown in FIG. 10); and when theoperation button 22D is pressed, the movable portion 23D moves downward((4) shown in FIG. 10).

Note that in the present embodiment, the operation buttons 22A and 22Dare located substantially at the same position as seen from theimage-capturing direction. Therefore, when viewed from theimage-capturing direction, the operation button 22D itself issubstantially invisible by being blocked by the operation button 22A.Thus, in the present embodiment, the movable portion 23D correspondingto the operation button 22D is offset from any of the other movableportions 23A to 23C (as viewed from the image-capturing direction). Thatis, the movable portions 23A to 23D are located so that they are visibleas viewed from the image-capturing direction, as shown in FIG. 10. Thus,the movement of the movable portion 23D following the pressing of theoperation button 22D can be recognized when viewed from theimage-capturing direction.

FIG. 11 shows an example captured image obtained by the infrared camera4. The captured image shown in FIG. 11 is an image to be obtained whenthe operation buttons 22A to 22D are not operated (in the unoperatedstate). As shown in FIG. 11, the captured image includes an image(reference marker image) 30 of the reference marker 21 d and images(movable portion images) 31A to 31D of movable portions 23A to 23D.Thus, the reference marker 21 d and the movable portions 23A to 23D areplaced at such positions as to be included in the image-capturing rangeof the infrared camera 4, with the operation device 20 attached to theportable device 1. Thus, with the operation device 20 connected to theportable device 1, the infrared camera 4 can capture an image of thereference marker 21 d and the movable portions 23A to 23D inside thehousing 21 through the hole 21 a of the housing 21. That is, thefollowing members (a) to (d) are placed at such positions that theinfrared camera 4 can capture an image of the reference marker 21 d andthe movable portions 23A to 23D in such a state as described above:

(a) infrared camera 4 of portable device 1;

(b) hole 21 a of housing 21 of controller device 20;

(c) reference marker 21 d of controller device 20; and

(d) movable portions 23A to 23D of controller device 20.

FIG. 12 shows another example captured image obtained by the infraredcamera 4. The captured image shown in FIG. 12 is an image obtained whilethe operation button 22A is operated (depressed). When the operationbutton 22A is operated, the movable portion 23A moves downward.Therefore, in the captured image, the movable portion image 31A islocated below its position in the unoperated state, as shown in FIG. 12.Thus, when an operation button 22 is operated, the position of thecorresponding movable portion image 31 changes from its position in theunoperated state. Therefore, based on the position of the movableportion image 31 in the captured image, it is possible to determinewhether the operation button 22 corresponding to the movable portionimage 31 is operated.

Specifically, the portable device 1 calculates the position of thereference marker image 30 from the captured image. Any method may beused for recognizing the reference marker image 30 in the capturedimage. For example, the portable device 1 identifies the referencemarker image 30 by performing an image recognition process using patternmatching on the captured image. Moreover, the portable device 1calculates the position of the identified reference marker image 30 (theposition on the captured image). In the present embodiment, thecalculated position of the reference marker image 30 is used as thereference position.

The portable device 1 calculates the position of each of the movableportion images 31A to 31D from the captured image. Any method may beused for recognizing the movable portion image in the captured image.The portable device 1 identifies each of the movable portion images 31Ato 31D by performing an image recognition process using patternmatching, for example. Moreover, the portable device 1 calculates theposition of each of the identified movable portion images 31A to 31D(the position on the captured image).

Note that the movable portion images 31A to 31D have the same shape inthe present embodiment, and it is therefore difficult to distinguish themovable portion images 31A to 31D from one another based only on theshape. Note however that the positional relationship between the movableportion images 31A to 31D (i.e., the order of arrangement from the left,i.e., the movable portion image 31C, the movable portion image 31A, themovable portion image 31B and the movable portion image 31D) ispredetermined and known. The positional relationship between each of themovable portion images 31A to 31D and the reference marker image 30 ispredetermined. Therefore, the portable device 1 can identify each of themovable portion images 31A to 31D by using either one of thesepositional relationships. Note that in other embodiments, therecognition portions of the movable portions may have different shapesfrom each other. Then, it is possible to identify each movable portionimage based on the shape of the movable portion image.

The portable device 1 determines whether or not the operation buttons22A to 22D have been operated based on the calculated positions of themovable portion images 31A to 31D. Now, if an operation button 22 isoperated, the distance from the reference marker image 30 to the movableportion image (the movable portion image corresponding to the operationbutton 22) becomes longer than that in the unoperated state. That is, itis possible to determine whether or not each of the operation buttons22A to 22D is operated (pressed) by determining whether or not thedistance between each of the movable portion images 31A to 31D and thereference position is longer than that in the unoperated state.

Thus, in the present embodiment, the portable device 1 determineswhether or not the distance from the reference position (the position ofthe reference marker 21 d) to each of the movable portion images 31A to31D is greater than or equal to a predetermined distance. Here, thepredetermined distance is set for each of the movable portion images 31Ato 31D, and is set to be longer than the distance from the referenceposition to the movable portion image in the unoperated state. If thedistance from the reference position to the movable portion image isgreater than or equal to the predetermined distance, the portable device1 determines that the operation button 22 corresponding to the movableportion image is operated. On the other hand, if the distance from thereference position to the movable portion image is shorter than thepredetermined distance, the portable device 1 determines that theoperation button 22 corresponding to the movable portion image is notoperated.

As described above, in the present embodiment, whether or not anoperation button is operated is determined based on the positionalrelationship between the reference position and the position of themovable portion image. Here, the specific method of determination may beany method, and the determination may be done by any other method. Forexample, in other embodiments, the determination may be done based onlyon the position of the movable portion image (without using thereference position). Specifically, the portable device 1 may pre-store,as a registered position, the position of the movable portion image inthe unoperated state, and may perform the determination based on whetheror not the position of the movable portion image in the captured imageobtained is displaced from the registered position by a predetermineddistance or more.

Note that the positional relationship between the portable device 1 andthe operation device 20 attached thereto is not always constant, but mayvary to some degree each time the operation device 20 is attached orbetween different operation devices. Therefore, the position of themovable portion image on the captured image is not always constant evenin the unoperated state, but may vary each time the operation device 20is attached or between different operation devices. Therefore, with themethod for performing the determination based only on the position ofthe movable portion image, there is a possibility that whether or not anoperation button is operated cannot be determined accurately because thepre-stored registered position is not correct. In contrast, thedetermination can be done accurately by using the method for performingthe determination based on the positional relationship between thereference position and the position of the movable portion image as inthe present embodiment.

Note that in other embodiments, the portable device 1 may reset(re-store) the registered position each time the operation device 20 isattached to the portable device 1, thereby performing the determinationaccurately. For example, when the operation device 20 is attached to theportable device 1, the portable device 1 may store the position of themovable portion image as the registered position while showing the usera message such as “Do not press buttons”. Then, it is possible to storea more accurate registered position.

As described above, in the present embodiment, based on the imagecaptured by the infrared camera 4, the portable device 1 can determinethat a movable portion 23 has moved and can determine whether or not theoperation button 22 is operated.

[5. Process of Portable Device]

Next, referring to FIG. 13 and FIG. 14, an example information processperformed by the portable device 1 will be described. FIG. 13 is a flowchart showing an example of the flow of a process performed by theportable device 1. A series of processes shown in FIG. 13 is performedby the CPU of the control section 14 executing a predetermined programstored in the portable device 1. In the present embodiment, thepredetermined program is a system program for running the OS (operatingsystem) of the portable device 1. Note however that in otherembodiments, the predetermined program may be any program, e.g., anapplication program to be run on the OS, for example. In the presentembodiment, the series of processes shown in FIG. 13 is repeatedlyperformed while the power of the portable device 1 is ON. Note howeverthat in other embodiments, the control section 14 may halt the series ofprocesses when the portable device 1 is in a sleep state (e.g., wherethe display 2 is OFF), while performing the series of processes when theportable device 1 is in an active state (e.g., where the display 2 isON). The control section 14 may activate the infrared camera 4 and theilluminating section 7 while the series of processes is performed, anddeactivate the infrared camera 4 and the illuminating section 7 whilethe series of processes is halted.

Note that in the present application, the process in each step in theflow chart shown in the figure is merely an example, and the order ofsteps may be switched around, or other processes may be performed inaddition to (or instead of) these steps, as long as similar results areobtained. While it is assumed herein that the processes of the steps ofthe flow chart are performed by the CPU of the portable device 1,processes of some of the steps of the flow chart may be performed by aprocessor or a dedicated circuit other than the CPU. Some of theprocesses performed by the portable device 1 may be performed by anotherinformation processing device capable of communicating with the portabledevice 1 (e.g., a server capable of communicating with the portabledevice 1 via a network). That is, the processes shown in FIG. 13, FIG.14 and FIG. 19 may be performed through a cooperation of a plurality ofinformation processing devices including the portable device 1.

In the series of processes shown in FIG. 13, first, in step S1, thecontrol section 14 obtains, from the infrared camera 4, a captured imageobtained by the infrared camera 4. That is, as the infrared camera 4captures an image at a predetermined time interval, the control section14 obtains the captured image from the infrared camera 4 and stores dataof the captured image in a memory.

In step S2, the control section 14 determines whether or not thereference marker 21 d is detected in the captured image obtained in stepS1. That is, the control section 14 performs an image recognitionprocess by using the method described in “[4. Determining operationperformed on operation device]” above. If the reference marker image 30is identified by the image recognition process, the control section 14determines that the reference marker 21 d is detected. On the otherhand, if the reference marker image 30 is not identified by the imagerecognition process, the control section 14 determines that thereference marker 21 d is not detected. If the determination result ofstep S2 is affirmative, the process of step S3 is performed. On theother hand, if the determination result of step S2 is negative, theprocess of step S1 is performed again. In this case, the series ofprocesses of steps S1 and S2 is repeatedly performed until it isdetermined in the process of step S2 that the reference marker 21 d isdetected.

In step S3, the control section 14 launches a predetermined application(program) stored in the portable device 1. The predetermined applicationmay be any application, and may be a game application, for example. Notethat the series of processes performed in the predetermined applicationwill be described later with reference to FIG. 14. When the execution ofthe predetermined application is ended, the control section 14 ends theprocess of step S3, and then performs again the process of step S1.

As described above, in the present embodiment, if the operation device20 is attached to the portable device 1 while performing the series ofprocesses of steps S1 to S3 described above, the reference marker 21 dis detected in the process of step S2, and the application is launchedautomatically (irrespective of user instructions) in the process of stepS3. Thus, the user can launch an application associated with theoperation device 20 by attaching the operation device 20 to the portabledevice 1. That is, the user does not need to perform an operation ofselecting and launching an application, and thus the user operation issimplified in the present embodiment.

Note that in other embodiments, there may be different types ofoperation devices each provided with a reference marker of a differentshape and/or size, and the control section 14 may launch a differentapplication depending on the type (the shape and/or size) of thereference marker detected in step S2 (see “(Variation in which processis varied depending on type of marker)” to be described later). That is,a different application may be launched in response to the attachment ofa different operation device provided with a different reference marker.

Although not shown in FIG. 13, the control section 14 may performvarious processes to be performed on the OS level. For example, thecontrol section 14 may perform processes such as the process ofdisplaying on the display 2 a menu image for launching an application,and the process of launching the application in response to the userspecifying the application on the menu image.

Next, referring to FIG. 14, a process example to be performed by theapplication launched by the process of step S3 will be described. FIG.14 is a flow chart showing an example of the flow of a process performedby an application program on the portable device 1.

In the series of processes shown in FIG. 14, first, in step S11, thecontrol section 14 performs a calibration process. Specifically, thecontrol section 14 calculates the reference position described above bythe method described in “[4. Determining operation performed onoperation device]” above, and stores the data of the reference positionin a memory. Note that the captured image used for the calculation ofthe reference position may be an image obtained at an arbitrary point intime after the operation device 20 is attached to the portable device 1.For example, the reference position may be calculated by using acaptured image in which the reference marker 21 d has been detected instep S2, or may be calculated by using a captured image that is obtainedfrom the infrared camera 4 at the point in time when the process of stepS11 is performed. The calibration process may be performed by the systemprogram before the launch of the application. For example, in otherembodiments, the calibration process may be performed before the processof step S3 when the determination result of step S2 is affirmative.

In step S12, the control section 14 obtains a captured image from theinfrared camera 4. In step S13 to follow, the control section 14determines an operation performed on the operation device 20 based onthe captured image obtained in step S12. That is, the control section 14determines whether or not an operation has been performed for each ofthe operation buttons 22A to 22D by the method described in “[4.Determining operation performed on operation device]” above. Note thatthe reference position used in the process of step S13 is calculated inthe process of step S11.

In step S14, the control section 14 performs a process in accordancewith the content of the operation as determined in step S13. Thespecific content of the process is not limited to any particularcontent, and the process may be a process of controlling a character inaccordance with the content of the operation in a case where a gameapplication is performed, for example. In step S14, the process may beperformed in accordance with operations performed on the operationsections (the input button 6 and the touch panel 3) of the portabledevice 1, as well as those performed on the operation buttons 22A to22D.

In step S15, the control section 14 determines whether or not to end theapplication. The specific method for the determination in step S15 maybe any method. In the present embodiment, it is determined to end theapplication when the user gives a predetermined ending instruction orwhen the operation device 20 is detached from the portable device 1.Specifically, the control section 14 performs a first determinationprocess of determining whether or not the user has given a predeterminedending instruction, and performs a second determination process ofdetermining whether or not the operation device 20 has been detachedfrom the portable device 1. For example, the second determinationprocess may be performed based on whether or not the reference marker 21d or the movable portions 23A to 23D has been identified from thecaptured image obtained in step S12. If the determination result ofeither the first determination process or the second determinationprocess is affirmative, the control section 14 determines to end theapplication. In such a case, the control section 14 ends theapplication. On the other hand, the determination results from the firstdetermination process and the second determination process are bothnegative, the control section 14 determines not to end the application.In such a case, the control section 14 performs again the process ofstep S12, and thereafter repeatedly performs the series of processes ofsteps S12 to S15 until it is determined to end the application in stepS15.

As described above, in the present embodiment, if the operation device20 is removed from the portable device 1 while the series of processesis performed in steps S12 to S15, the process of step S15 ends theapplication. Therefore, the user can end the application using theoperation device 20 simply by removing the operation device 20 from theportable device 1. That is, the user does not need to perform anoperation of ending an application, and thus the user operation issimplified in the present embodiment.

[6. Function and Effect of the Present Embodiment and VariationsThereof]

As described above, in the embodiment above, the operation device 20 canbe attached to an information processing device (the portable device 1)including an image-capturing unit (the infrared camera 4), and has thefollowing configuration:

-   -   a housing 21 of which the inside is visible from the outside        through at least a portion thereof (a portion where the opening        portion (the hole 21 a) is formed);    -   a movable operation section (the operation button 22) at least a        portion of which is exposed on the outside of the housing 21;        and    -   a movable portion 23 placed at a position inside the housing 21        that is visible from the outside of the housing 21 through the        above portion of the housing 21.

The operation device 20 is attached to the information processing deviceat such a position that the image-capturing unit can capture an image ofthe inside of the housing 21 through the opening portion (see FIG. 8).The information processing device determines an operation performed onthe operation section based on a captured image (a captured imageincluding an image of the movable portion) obtained by theimage-capturing unit (step S13), and performs a predetermined processbased on the determined operation (step S14).

With the configuration described above, the movable portion 23 maychange its position, may change its attitude, or may change its positionand its attitude in response to the operation section being operated.The method for determining an operation performed on the operationsection based on the captured image including the image of the movableportion 23 may be a method for determining an operation based on atleast one of the position, the direction and the shape of the movableportion 23 in the captured image, for example.

With the above configuration, the information processing device to whichthe operation device 20 is attached can capture an image of the movableportion by means of the image-capturing unit, and determine an operationperformed on the operation section based on the captured image.Therefore, it is possible to simplify the configuration of the operationdevice because the operation device does not need an electricalconfiguration for detecting the operation and transmitting the detectionresult to the information processing device.

Note that the information processing device to which the operationdevice is attached is not limited to a device including a displaysection as in the embodiment above, but may be a controller device suchas a game controller, for example. For example, the game controller maybe a game controller for capturing an image of an outside marker bymeans of a camera, and the operation performed on the game controller(e.g., an operation of pointing at the screen, or an operation of movingthe game controller) may be determined based on the captured image. Anoperation device such as that of the embodiment above may be attached tosuch a game controller, and the camera of the game controller maycapture an image of the movable portion of the operation device. Then,it is possible to determine the operation performed on the operationsection based on the captured image obtained by the camera. Theoperation device may be attached to an information processing systemincluding a controller, and a main console device that can beattached/detached to/from the controller. For example, the operationdevice may be attached to a portable information processing device(which can also be said to be an information processing system)including a main console device and a controller. For example, thecontroller may include the image-capturing unit, and determine anoperation performed on the operation section based on the captured imageobtained by the image-capturing device. Then, the main console devicemay perform a predetermined process based on the determined operation.

Note that it is not necessary that the operation device 20 be a deviceincluding no electrical configuration. In other embodiments, theoperation device 20 may include an electrical configuration such as anelectronic circuit. For example, the operation device 20 may include acommunication section for communicating with another informationprocessing device different from the portable device 1 to which it isattached. The operation device 20 may be electrically connected to theportable device 1, and may for example include a speaker for receiving asound signal from the portable device 1 to output sounds. Even in thecase of an electrical configuration as described above, it is possibleaccording to the embodiment above to simplify the configuration of theoperation device regarding the configuration for the operation section.

Note that the infrared camera 4 is used as the image-capturing unit forcapturing an image of the movable portion in the embodiment above. Here,the image-capturing unit may be any image-capturing device (i.e., acamera) capable of capturing an image. In other embodiments, a visiblelight camera (a camera using a visible light image sensor, e.g., an RGBcamera) may be used as the image-capturing unit. In such a case, theilluminating section 7 may be a flash provided in the informationprocessing device used when capturing an image.

Note that the movable portion may include a recognition portion in anarea that is visible from the outside of the housing through a portionof the housing 21 (the portion of the hole 21 a), wherein therecognition portion is distinguishable from other portions (movableportions other than the identification portion and the inner wallsurface of the housing). Note that “distinguishable” means that it canbe distinguished by an image-capturing device (the infrared camera 4).For example, the movable portion may include, in the area, anidentification portion that is different from other portions in terms ofat least one of the shape, the design pattern, the color and thematerial. Then, the image-capturing unit can easily recognize theidentification portion as distinguished from other portions.

In the embodiment above, at least a portion of the area of the movableportion 23 (the hatched area shown in FIG. 7) that is visible from theoutside of the housing 21 through the opening portion of the housing 21is formed by a member that can be distinguished by the infrared camera 4from other portions. Then, it is possible to capture an image of themovable portion by using an infrared camera, and determine an operationperformed on the operation section based on the captured image.

In the embodiment above, the operation section includes the operationbutton 22A and the operation button 22D that can be operatedindependently of each other. The operation button 22A and the operationbutton 22D are arranged in line with each other in the front-reardirection as seen through the opening portion (see FIG. 7). The movableportion includes the movable portion 23A of which the position (or theattitude) changes in response to the operation button 22A beingoperated, and the movable portion 23D of which the position (or theattitude) changes in response to the operation button 22D being operated(see FIG. 7). Here, the movable portion 23D is displaced from themovable portion 23A as seen through the opening portion (see FIG. 10).Then, even if a plurality of operation buttons are arranged in line witheach other in the front-rear direction as seen from the image-capturingunit of the information processing device, it is possible to determineoperations on the operation buttons by capturing an image of the movableportions.

In the embodiment above, on the inside of the housing 21, apredetermined reference marker 21 d is provided at a position inside thehousing 21 that is visible from the outside of the housing 21 throughthe opening portion of the housing 21. The position at which the movableportion is provided is different from the position at which the markeris provided (see FIG. 6). The information processing device recognizesthe reference marker 21 d in the captured image to determine anoperation performed on the operation button 22 based on the position ofthe image of the movable portion 23 with respect to the reference marker21 d (the position of the image of the reference marker 21 d) in thecaptured image. Note that the determination of the operation can be donebased at least on one of the position, the direction and the shape ofthe image of the movable portion 23 with respect to the reference marker21 d.

Then, the information processing device can more accurately detect theposition, the direction and/or the shape of the movable portion 23, andit is therefore possible to accurately determine an operation.

(Variation Regarding Operation Section of Operation Device)

In other embodiments, the operation section of the operation device 20may include an operation means other than an operation button. Forexample, the operation section may be a rotational operation sectionthat can be rotated. Referring to FIG. 15 and FIG. 16, a variation wherethe operation device 20 includes a rotational operation section as anoperation section will now be described.

FIG. 15 shows an example rotational operation section of the operationdevice 20. In the present variation, the operation device 20 includes arotational operation section 41 that can be rotated by the user.Specifically, the rotational operation section 41 is provided on theoutside of the housing 21, and is provided rotatably relative to thehousing 21. Note that although the rotational operation section 41 canrotate about the z axis in FIG. 15, the direction of rotation may be anydirection in other embodiments.

As shown in FIG. 15, the rotational operation section 41 is fixedlyconnected to a rotation shaft 42. A rotatable portion 43 is placedinside the housing 21, and the rotation shaft 42 is connected to therotatable portion 43 via, for example, a gear (not shown). Although notshown in the figure, the rotatable portion 43 is supported inside thehousing 21 so that the rotatable portion 43 can rotate in accordancewith the rotation of the rotation shaft 42. Although the rotatableportion 43 is configured so that the rotatable portion 43 can rotateabout the y-axis direction in FIG. 15, the direction of rotation may beany direction in other embodiments (see FIG. 16). The rotatable portion43 includes a recognition portion (hatched area shown in FIG. 15) whosedirection changes by being rotated.

Also in the present variation, as in the embodiment above, the portabledevice 1 can determine an operation performed on the operation sectionbased on the captured image obtained by the infrared camera 4.Specifically, the control section 14 of the portable device 1 capturesan image of the rotatable portion 43 by means of the infrared camera 4,thereby obtaining a captured image including the image of the rotatableportion 43. Then, the control section 14 identifies, from the capturedimage, the image of the recognition portion of the rotatable portion 43,and calculates the angle of rotation of the rotational operation section41 based on the identified image direction. The angle of rotation to becalculated may be an absolute value or a relative value relative to apredetermined reference (e.g., an angle of rotation calculated from thepreviously-obtained captured image). The portable device 1 performs apredetermined process using the calculated angle of rotation as anoperation input.

Note that the mechanism of the rotatable portion 43 for calculating theangle of rotation of the rotational operation section 41 is not limitedto the one described above, but may be any other mechanism. For example,in other embodiments, the rotatable portion connected to the rotationaloperation section 41 may be rotatable in the same rotation direction asthe rotational operation section 41. FIG. 16 shows another examplerotational operation section of the operation device 20. In FIG. 16, arotatable portion 45 is connected to the rotational operation section 41and the rotation shaft 42 so that they rotate about the same rotationaxis. That is, the rotatable portion 45 rotates by the same amount inthe same direction as the rotational operation section 41 in response tothe rotational operation section 41 being rotated.

In FIG. 16, the rotatable portion 45 has a cylindrical shape. Here, arecognition portion is formed on the side surface of the rotatableportion 45 so that its height (length in the z-axis direction) variesdepending on the angle of rotation (see the unfolded diagram of the sidesurface shown in FIG. 16). Then, in the captured image obtained by theinfrared camera 4, the height of the image of the recognition portion ofthe rotatable portion 45 (e.g., the height of the image of therecognition portion at the central position in the left-right direction)varies depending on the angle of rotation. Therefore, it is possible tocalculate the height based on the captured image, and to calculate theangle of rotation of the rotatable portion 45, i.e., the angle ofrotation of the rotational operation section 41, based on the height.

As described above, the recognition portion is formed on the sidesurface of the rotatable portion 45 so that the recognition portionappears to have a different shape (or a different design pattern)depending on the angle of rotation. Note that the shape of therecognition portion is not limited to the shape shown in FIG. 16. FIG.17 shows another example of an unfolded diagram of the side surface ofthe rotatable portion 45. In other embodiments, a design pattern asshown in FIG. 17 may be formed on the side surface of the rotatableportion 45 as the recognition portion. Then, the control section 14 ofthe portable device 1 detects the design pattern (pattern) of therecognition portion in a predetermined area (e.g., the central area inthe left-right direction) of the image of the rotatable portion 45included in the captured image. The control section 14 can calculate theangle of rotation of the rotatable portion 45, i.e., the angle ofrotation of the rotational operation section 41, based on the detectedpattern.

As described above, in the variation above, the operation section isconfigured to be rotatable. The movable portion (the rotatable portion43, 45) rotates in accordance with the rotation of the operationsection, and the shape and/or the design pattern of the movable portionas seen through the opening portion changes as the movable portionrotates. Then, based on the captured image (more specifically, based onthe shape and/or the design pattern of the movable portion in thecaptured image), the portable device 1 determines an operation performedon the operation section (the angle of rotation). Then, the portabledevice 1 can determine an operation performed on the operation section,which is operated by being rotated by the user. That is, it is possibleto use, as an operation device attached to the portable device 1, anoperation device having an operation section that is operated by beingrotated by the user.

In other embodiments, the operation section may be a stick that can betilted in the up, down, left and right directions, or may be a slide padthat is slidable in the up, down, left and right directions. In the caseof a stick or a slide pad, the movable portion is configured so that themovable portion moves in a direction depending on the tilt direction orthe slide direction. Thus, it is possible to determine the tiltdirection or the slide direction based on the image of the movableportion.

Note that although the movable portion 23 moves in the up-downdirection, as seen through the opening portion (i.e., as seen from theimage-capturing direction), in response to the operation section beingoperated in the embodiment above, the moving direction of the movableportion may be any other direction. For example, the movable portion maymove in at least one of the left-right direction and the front-reardirection as seen from the image-capturing direction.

Note that the position (movement) in the up-down direction and theleft-right direction can be determined based on the position (theposition on the captured image) of the image of the recognition portionof the movable portion. The position (movement) in the front-reardirection can be determined based on the brightness value of the image(i.e., the value representing the brightness of each pixel in thecaptured image) of the recognition portion. That is, where theilluminating section 7 outputs infrared light, as the recognitionportion is positioned more forward, the brightness value will be higherin an area of the recognition portion, and the portable device 1 cantherefore determine the position of the movable portion in thefront-rear direction based on the brightness value of the area. In theembodiment above, the position (movement) in the front-rear directionmay be calculated by using detection results from the distance measuringsensor 5.

In the embodiment above, the portable device 1 determines whether or notthe operation section is operated based on the captured image, andperforms a predetermined process based on whether it is determined thatthe operation section has been operated. Note that although the portabledevice 1 performs a predetermined process in response to the operationsection being operated in the embodiment above, the portable device 1may, in other embodiments, perform a predetermined process in responseto an operation on the operation section being released (e.g., therelease of pressing of an operation button). In the variation shown inFIG. 15, the portable device 1 calculates the amount of operation on theoperation section based on the captured image, and performs apredetermined process based on the amount of operation. Thus, for anoperation performed on the operation device 20, the portable device 1may calculate a digital input value (ON/OFF) or calculate an analoginput value (an input value represented in three or more steps).

Note that even if the operation device 20 of the embodiment above isused, the portable device 1 can calculate an analog input value. Thatis, in the embodiment above, it is possible to calculate the amount ofoperation (amount of pressing) on the operation button 22 depending onthe distance from the reference position described above to the movableportion image.

(Variation in which Process is Varied Depending on Type of Marker)

Next, a variation in which the portable device 1 varies the process tobe performed based on the type of the marker provided on the operationdevice. In the present variation, a process marker is provided on theinner wall surface of the housing 21 of the operation device 20. Aprocess marker is a marker representing the content of the process to beperformed when the process marker is detected. In the present variation,when a process marker is identified from the captured image, theportable device 1 performs a process in accordance with the processmarker. Referring to FIG. 18 and FIG. 19, the details of the presentvariation will be described.

FIG. 18 shows an example captured image obtained by the infrared camera4 according to the present variation. As shown in FIG. 18, according tothe present variation, when the infrared camera 4 captures an image ofthe inside of the housing 21, the captured image includes an image 35 ofa process marker, in addition to the reference marker image 30 and themovable portion images 31A to 31D. The shape of the process marker maybe any shape. For example, the process marker may be a two-dimensionalbar code.

FIG. 19 is a flow chart showing an example of the flow of a processperformed by the portable device 1 according to the present variation.According to the present variation, a series of processes shown in FIG.19 is performed, instead of the series of processes of the embodimentabove shown in FIG. 13. The flow of the process of the present variationwill now be described while focusing on its differences from the seriesof processes shown in FIG. 13.

Also in the present variation, as in the embodiment above, the processesof steps S1 and S2 are performed. In the present variation, the controlsection 14 performs the process of step S21 if the determination resultof step S2 is affirmative. On the other hand, if the determinationresult of step S2 is negative, the process of step S1 is performed againas in the embodiment above.

In step S21, the control section 14 determines whether or not a processmarker has been detected from the captured image obtained in step S1.The specific method for the determination process of step S21 may besimilar to the determination process of step S2. If the determinationresult of step S21 is affirmative, the process of step S22 is performed.On the other hand, if the determination result of step S21 is negative,the process of step S1 is performed again.

Note that in other embodiments, the operation device 20 may not beprovided with the reference marker 21 d and be provided only with aprocess marker. Then, the portable device 1 may perform the process ofstep S21 following step S1.

In step S22, the control section 14 performs a process in accordancewith the type of the marker detected in step S21. Here, the method foridentifying the process associated with the detected marker may be anymethod. For example, the portable device 1 may pre-store a table inwhich a plurality of types of markers are each associated with a processto be performed when the marker is detected. Then, in step S22, theportable device 1 identifies the process associated with the detectedmarker with reference to the table. For example, where the processmarker is capable of representing information (e.g., where it is atwo-dimensional bar code), information for identifying the process to beperformed when the process marker is detected may be included in theinformation represented by the process marker. Then, the portable device1 analyzes the detected marker to read information, and identifies, asthe process to be performed, the process indicated by the readinformation.

The process to be performed in accordance with the type of the detectedmarker may be of any content. The process performed in step S22 may be,for example, a process of launching a predetermined application programwith which the operation device 20 can be used. The process performed instep S22 may be a process of obtaining a predetermined applicationprogram with which the operation device 20 can be used. That is, thecontrol section 14 may access a predetermined external device (e.g., aserver capable of communicating with the portable device 1 via anetwork) to obtain the application program. Then, the obtainedapplication program may be executed automatically. The process performedin step S22 may be a process of obtaining a web page from a server anddisplaying the web page on the display 2. For example, the web page tobe displayed may be a web page for downloading an application program,or an advertisement of the vendor of the operation device 20.

Note that the control section 14 may change the type of the process tobe performed depending on the type of the marker detected. That is,depending on the type of the marker detected, the control section 14 mayperform one of the process of launching an application program, theprocess of obtaining an application program, and the process ofdisplaying a web page. Depending on the type of the marker detected, thecontrol section 14 may change the content of the process within the sameprocess type. That is, depending on the type of the marker detected, thecontrol section 14 may change the application program to be launched,change the application program to be obtained, or change the web page tobe displayed.

In step S22, when the process in accordance with the type of the markeris ended, the control section 14 ends the process of step S22, and thenperforms again the process of step S1.

As described above, according to the variation above, the portabledevice 1 performs a different process depending on the type of theprocess marker recognized based on the captured image. Then, byproviding a different process marker for each operation device, it ispossible, when an operation device is attached to the portable device 1,for the portable device 1 to perform a process in accordance with theoperation device.

As described above, according to the embodiment above and the variationabove, inside the housing 21, a predetermined marker (the referencemarker 21 d or a process marker) is provided at a position inside thehousing 21 that is visible from the outside of the housing 21 through anopening portion of the housing 21. The information processing device(the portable device 1) recognizes the predetermined marker in thecaptured image obtained by the infrared camera 4 (step S2, S21), andperforms a predetermined process based on the marker recognition result(step S3, S15, S22).

Then, since the portable device 1 performs a predetermined process basedon the marker recognition result, an appropriate process in accordancewith an operation device attached to the portable device 1 can beperformed without the user giving an instruction to perform thepredetermined process. This makes it possible to save the trouble of theuser having to give an instruction, and to reduce the possibility thatan inappropriate process may be performed due to an erroneous operationby the user, or the like. This advantage can be realized not only whenthe accessory is an operation device, but also with any accessory thatadds a function to the information processing device (portable device).That is, the accessory is not limited to an operation device that addsan operation function to the information processing device, but may alsobe a device that adds a predetermined function.

The “marker recognition result” may be the determination resultindicating whether a marker has been recognized, or may be the result ofidentifying the type of the marker recognized. That is, the portabledevice 1 may perform a predetermined process (a process of launching anapplication, or a process of ending an application) based on thedetermination result indicating whether or not a marker has beenrecognized, as in the embodiment above. Alternatively, the portabledevice 1 may perform a predetermined process (a process performed instep S22) based on the result of identifying the type of the markerrecognized, as in the variation above. The portable device 1 may becapable of recognizing a certain type of a marker, or may be capable ofrecognizing a plurality of types of markers.

For example, the “predetermined process” may be a process as follows.That is, the portable device 1 may determine that the operation device20 has been attached to the portable device 1 when it is determined thatthe predetermined marker has been recognized. Note that when it isdetermined that the operation device 20 has been attached to theportable device 1, the portable device 1 may give the user anotification or may launch a predetermined application (as in theembodiment above).

Alternatively, if it is determined that the predetermined marker hasbeen recognized, the portable device 1 may determine that an authenticoperation device has been attached to the portable device 1 and allow apredetermined process to be performed on the portable device 1. If it isdetermined that the predetermined marker cannot be recognized, theportable device 1 may determine that an unauthentic operation device hasbeen attached to the portable device 1 and prohibit the predeterminedprocess from being performed. For example, the predetermined process maybe the process performed in step S22 described above, or may be anyother process.

As described above, the embodiment above can be applied to a portabledevice, such as a mobile telephone, a smartphone or a portable gamedevice, and to an accessory (attachment) that can be attached to such aportable device, with the aim of providing an accessory with a simpleconfiguration.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. An information processing system including aninformation processing device having a camera, and an accessory to whichinformation processing device is removably attachable, wherein: theaccessory includes: a housing having an opening; a movable operationsection at least a portion of which is exposed on the outside of thehousing; and a movable portion placed at a position inside the housing,wherein a position and/or an attitude of the movable portion changes inresponse to the movable operation section being operated; theinformation processing device is attached to the accessory at such aposition that the camera can capture an image of the inside of thehousing through the opening; and one or more processor of theinformation processing system: determines an operation performed on themovable operation section based on a camera image captured by the cameraand including an image of the movable portion; and performs apredetermined process based on the determined operation.
 2. Theinformation processing system according to claim 1, wherein: theprocessor determines whether or not the movable operation section isoperated based on the camera image; and the processor performs apredetermined process based on whether or not the movable operationsection is operated.
 3. The information processing system according toclaim 1, wherein: the processor calculates an amount of operationperformed on the movable operation section based on the camera image;and the processor performs a predetermined process based on the amountof operation.
 4. The information processing system according to claim 1,wherein: on the inside of the housing, a marker is provided at aposition inside the housing, wherein the position at which the movableportion is provided is different from the position at which the markeris provided; and the processor recognizes the marker in the camera imageto determine an operation performed on the moveable operation sectionbased on at least one of a position, a direction and a shape of an imageof the movable portion with respect to the marker in the camera image.